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ABSTRACT 

Prepared for use with Research for Better Schools! 
career education programs, this guide is designed to assist in the 
selection cf instruments which measure program goals. The guide 
provides descriptions of ten instruments: (1) student demcgraphic 
data questionnaire, (2) career maturity inventory, (3) assessment of 
career development, (4) self-directed interest inventory, (5) 
assessment of student attitudes toward learning envircnmgents, (6) 
student attitude survey, (7) ccmprehensive tests of basic skills, (8) 
student opinion survey, (9) parent opinion survey, and (10) community 
participant opinion survey. Each instrument is described in terms of 
its rationale and objectives, ccntent and organization, general 
administration procéiures, scoring procedures, scoring interpretation 
and use, psychometric qualities, and availability. The final section 
presents the intercorrelatione among each of the instruments. 
(LRA) 
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RESEARCH -FOR BETTER SCHOOLS, INC. (RBS), is a private, non-profit 

educational research laboratory located in Philadelphia, Pennsylvania. The INSTRU- 

MENT SERVICE GUIDE is part of a series of curriculum and procedural materials 

developed by the RBS CAREER EDUCATION PROGRAM (Michaelita B. Quinn, 
Director) for a pilot project in experience-based career education (EBCE). 

Additional materials in the evaluation series include: 


EVALUATION PLANNING MANUAL 
ANALYSIS SERVICE GUIDE 
PROGRAM MONITORING MENUBE 


RBS CAREER EDUCATION: INSTRUMENT SERVICE GUIDE was prepared a 
Thomas Biester : 
©Research for Better Schools, Inc. 


April 1977 
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INTRODUCTION 


Evaluation has been a continuing activity in the development of RBS Career 
Education. Evaluation findings have provided a useful source of information in both 
refining the program and in demonstrating evidence of program effectiveness to 
participants, sponsors, potential adopters and other members of the educational 
community. 

Since RBS Career Education has become available for adoption by public 
school districts, a series of materials and services has been prepared to assist adopters - 
in its planning, implementation and evaluation. This series includes: 


Materials Services 
Evaluation Planning Manual — Evaluation Technical Assistance 
Instrument Service Guide. — Instrument and Scoring Service : 
Analysis Service Guide — Analysis Service 
Program Monitoring Manual > = Evaluation Technical Assistance 


The materials assist in evaluation planning and design, while the services offer 
Research for Better Schools’ evaluation systems and expertise to support implemen- 
tation. Materials and services available from Research for Better Schools are 
described more completely in the RBS Evaluation Package Overview. 

The RBS Instrument Service Guide provides information about selecting 
program evaluation instruments. The selection of instruments which reasonably can 
be expected to measure the hypothesized program effects is a critical step in 
evaluation studies. For each hypothesis at least one measure must. be selected to 
represent the intended outcome. Thus, the selection of instruments must be tailored 
to the particular goals of the program being evaluated. 

"The RBS Career Education Program includes three general areas of student 
outcomes. These areas are (1) Career Skills, (2) Life or Self Skills and (3) Basic 
Academic Skills...Program implementers also need basic demographic information 
about students who are participating in the Career Education Program or serving as 
control group i ae Additionally, participant opinions can provide valuable 
feedback on perceived effectiveness of program elements. A series of instruments 
that measure the impact of programs similar to RBS Career Education is available. 
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lubeuniente Asia or constructed for each area of measutement are c listed 


below. 

© Career Skills . 
— Career Maturity Inventory 
— Assessment of Career Development 


— Student Attitude Survey (Career Attitude Scale) 
— Self-Directed Interest Inventory 
-@ Life or Self Skills 
— Assessment of Student Attitudes Towa Learning Environments 
— Student Attitude Survey (Acceptance of Self Scale) 
'— Student Attitude Survey (Acceptance of Others Scale) | : .% 


© Basic Academic Skills ee 
‘ - — Comprehensive Tests of Basic : Skills (Reading ‘Comprehension Test, 
Arithmetic Concepts Test, Arithmetic Applications Test) 


° Student Demographic Characteristics 
__— Student Demographic Data Questionnaire ; 


e Participant Opinion 
‘> — Student Opinion Survey 
— Parent Opinion Survey 
+ Community Participant Opinion Survey 


The RBS Instrument Service Guide describes each of the above instruments in 
terms of its rationale and objectives, content and organization, general administra- 
tion procedures, scoring procedures, scoring interpretation and use, psychometric 
qualities and availability, The intercorrelations among instruments are also 
presented. Instruments, directions for test administrators, technical assistance and 
scoring services are available from Research for Better Schools. 


INSTRUMENTS 


This section outlines the essential features of each instrument included in the- 
RBS Career Education Evaluation Package. These instruments are: 


. Student Demographic Data Questionnaire - 
Career Maturity Inventory 
_ Assessment of Career Development 
Self-Directed Interest Inventory 
Assessment of Student Attitudes Tawar? Learning Environments 
Student Attitude Survey ee * 
Comprehensive Tests of Basic Skills * 7, .* 
Student Opinion Survey - > 
Parent Opinion Survey 
Community Participant Opinion Survey 
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STUDENT DEMOGRAPHIC DATA QUESTIONNAIRE (SDQ) 


Ratioyale/Objectives of the Instrument: This form was designed to collect 
information on background and demographic characteristics of secondary 
school students and their parents and on the students’ career plans. 


Content and Organization: The SDQ is presented on a two-sided optical 
scanning form. Items include name, address, telephone number, birthdate, 
sex, ethnic group, grade level, school attendance, school GPA, reasons for 
applying to the career education ‘program, parental - -occupation” and 
student educational’ and occupational plans. 


Administration Procedures: The instrument is amenable to individual or 
group administration. Students should be given general instructions for 
completing optical scanning forms and be directed to complete all items: 
Time for completion is approximately 15 minutes. Comparison group 
students can omit the item on reasons for applying to the career education 
program. The SDQ should be administered to each student<during each 
year of program participation to update background information. é 


Scoring Procedures: Individual item scores are discrete. Occupational and 
educational scales can be combined to form a socioeconomic status 
estimate. 


Scoring Interpretation and Use: 


Responses are used for descriptive 
purposes and demographic input analysis. On an individual student basis, 
they provide background data useful in producing student profiles. The 
data can be combined across students to describe student groups by cate- 
' gorically interpreting items by frequency and percentage. Results of group: 
analyses have implications for the data analysis plan. For example, one can 
determine the extent to which experimental and control groups are 
matched according to demographic: variables or whether a posttest group 
is representative of the initial student sample. : 
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Psychometric Qualities: Content validity (in terms of usefulness)-has been 
established during a three-year evaluation effort by RBS and others.. 
Reliability.has not been formally determined. 


Availability: The Student Demographic Data Questionnaire and proces- 
sing services are available from Research for Better Schools, Inc. 


’ B. CAREER MATURITY INVENTORY (CMI) — COMPETENCE TEST 


ae 


Rationale/Objectives of the Instrument: This instrument was designed by 
John Crites to measure a set of cognitive competencies which reflect the 
author’s concept of career decision-making, These competencies are: 
Self-Appraisal, Occupational Information, Goal Selection, Planning and 
Problem Solving. Critical reviews by/RBS project staff and assessments by 
other researchers suggest that a subtests are most appropriate for 
assessing the effects of career education programs similar to RBS’. They 
are “Part 2: Knowing About Jobs”’ (Occupational Information) and “Part 
4: Looking Ahead” (Career Planning). Information on the other subscales 
can be abtained from the CMI Theory and Research Handbook. 


Content. and: Organization: All subtests are included in the CMI booklet; 
separate optical-scanning answer sheets are available. Items were written at 
a sixth grade reading level and should be appropriate for all secondary ’ 
school students. There are 20 multiple choice items in each subtest. The 


Occupational Information subtest assesses students’ knowledge of the 


world of work. Vignettes of workers at their jobs, based on worker trait 
requirements given in the Dictionary of Occupational Titles, constitute 
the stems for items. The task is to choose which occupational title 
matches the job description. The Planning subtest assesses students’ ability 
to plan the achievement of career goals. The item stems provide career 
goals of hypothetical individuals and sets of unordered steps to their 
attainment. The task is to.choose which sequence constitutes the correct 


order of stéps. 


Administration Procedures: The instrument can be administered to groups 
as well as to individuals. The CMI Administration and Use Manual gives 
complete instructions for administering all subtests in a standardized 
format. There is a suggested time limit of 20 minutes per subtest. It is 
recommended that this time limit be adhered to if the results are to be 
used for program evaluation. : 


Scoring Procedures: The score for each subtest is the number of items 
correctly answered. There is no overall score for the CMI. The CMI can be 
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scored ‘eiehies by had, of faachisi Keys ‘i odin ‘athe tue fiatd 
scoring are available from CTB/McGraw-Hill. Appropriate answer sheets, 


_ can be machine scored by: ‘either RBS or CTB/McGraw-Hill. vo 


Scdaitg Interpretation’ and Use: The higher the score on the CMI, che 
greater a stident’s knowledge in ‘ that area, Percentile norms by grade are 
presented in the CMI Administration and Use “Manual, but since the -- 
norming sample was relativély small, the development of Iocal norms is’. 
encouraged. : ae 


Phe CMI Manual ‘lists fens Ge sal uses «oF results: ‘study ing career 
development, assessing curricular and guidance ‘needs, evaluating career 


,education programs- ahd career counseling. Group scores. would be 
‘ appropriate for the first three uses, while individual student data would be 
_ used for the last. For evaluating career.education programs, pretest and 


posttest. results should be compared to determine student gains. Bless 
comparison with control group students is recommended. . 


Psychometric Quilities: Items were selected to discriminate between 


students of different grade levels. A significant monotonic linear trend was 
demonstrated for all items. Difficulty indices for many 9f the items in the 
Occupational Information subtest are very high, indicating the possibility . - 
of a ceiling effect for many students. Internal consistency coefficients of 
the Occupational Information ‘and Planning subtests are high, ranging. - 
from .81 to .90. Thus, the subtests are composed of rather homogeneous 
sets of items. Minimal validity data is reported in the CMI Theory and’ 
Research Handbook. ‘Reviews by RBS evaluators. suggest aia subtests 2 
and 4 appear to pense content validity. | : 


‘Availability: CMI test booklets, answers sheets and accessories are 


available through. CTB/McGraw-Hill. Processing for Digitek\answer sheets 
js available from RBS. Order forms and descriptive information.can be ~ 
obtained from RBS, 


ue 


C. ASSESSMENT OF CAREER DEVELOPMENT (ACD) 


Rationale/Objectives of the Instrument: This instrument was developed 
by the American College Testing Program. It focuses on three core 
components of career development: occupational awareness including 
occupational knowledge and exploratory occupational experiences; self 
awarness including job values and preferences, career plans, self-evaluation 
of career planning, and perceived needs for help with career planning; and 
career planning and decision-making including career planning knowledge 


__ and involvement in career planning experiences. The major purpose of the 


ACD is to provide counselors, administrators and evaluators with 


information to. develop guidance prerene and to, assess the outcomes of 


career guidance PESEE ae 


Content and Organization: The instrument is presented in booklet format 


_ with separate optical-scanning answer sheets available. The mean reading 
_ grade level is 7.2; the test is recommended for grades 8 through 11. The 


ACD is arranged into six units (subtests), two covering career-related 
knowledge and four covering career-related activities. Provisions are made 
for including up to 19 locally constructed items. Unit 1, Job Knowledge 
and Unit 5, Career Planning Knowledge are recommended for assessing 
career education program effeets. These two subtests measure students’ 
knowledge of occupational duties, psychosocial aspects, worker attributes, 
preparation requirements, reality factors, the career planning process and 
basic career development principles. Item formats are varied. 


Administration Procedures: The ACD is a group-administered test. 


Standardized administration procedures are given in the Test Administra- 


tor’s Manual of Instructions. The ACD Handbook suggests two formats 
for administering the instrument. One format involves three 45 minute 
periods while the other involves one 125 minute period. The two career 
knowledge subtests (Units 1 and 5) require 30 and 20 minutes, 


respectively, 


Scoring Procedures: The ACD must be scored iheauik the Houghton 
Mifflin Scoring Service. However, it is ‘possible to score the two career 


bites subtests by liand using the key in the ACD Handbook. The: 
Houghton Mifflin Scoring Service provides a Group Summary Report and. 
a Student List Report. The former.includes score distributions, means and °° 
standard deviations far 11 ACD scales and summaries of student responses 
to 42 specific items, The latter includes each individual student’ s score for 
the 11 scales and individual item responses to 19 questions. - \ 


Scoring Interpretation and. Use: ACD results can be used to obtain an 
overview of student plans, needs, and feelings; to identify students for” 
priority attention; to identify potential participants for special guidance . 
experiences; and to ‘assess students’ knowledge. about careers and career 
planning. Students’ career knowledge can be interpreted in a norm- 
referenced or criterion-referenced sense. The ACD- Handbook provides 
specific guidelines for the interpretation and use of results. E 


. Psychometric Qualities: The ACD appears to}phave been carefully 
constructed through a three stage empirical tryout ‘process. Item selection 
techniques included logical critiques by exptrts and item analysis data 
such as response distributions, difficulty indices and item-scale correla- 
tions. National norms were obtained on a sample of approximately 30,000 

. students, Validity data is reported in terms of content validity, scale . 
interrelationships, relationships to other variables and studies of career 
guidance outcomes and growth. Reliability estimates for the various ACD 
subscales range from: .61 to -.93. The reliability of the “Knowledge of 
Occupational Preparation Requirements” subscale is fairly low (.61); 
caution should be applied in the interpretation of individual scores. 


Ordering Information: ACD test booklets, answer sheets and accessories 
are available from tte Houghton Mifflin Company. ~ 
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D. . SELF-DIRECTED INTEREST INVENTORY (SDI). =} \. 5 


Rationale/Objectives of the’Instrument: This instrument is an adaptation 
of the Self-Directed Search* (SDS) developed by John Holland. The SDII 
is based on a theory of personality types and environmental models. The 
personal assessment and occupational classification systems use six major 
scales: Realistic, Investigative, Social, Conventional, Enterprising and 
Artistic. The SDII measures occupations considered; activities preferred, 
perceived competencies, occupations of interest and self-estimates of 
abilities. Primary differences between the SDII and the SDS are directions 
to students, format of presentation and availability of machine scoring, 


_The SDII is used with permission of Consulting Psychologists Press. 


\ 


Content and Organization: Three kinds of materials are required: an \ 


assessment booklet, an optical scanning response shett and a classification 
booklet. There are five sections in the assessment booklet: Occupations 
Considered, Activities (six scales of eleven items each), Competencies (six 
scales of eleven items each), Occupations (six scales of fourteen items 
each), and Self Estimates (two sets of six ratings). In the first section 
students are asked to list those occupations which they have considered in 
their career planning. The Activitjes scales estimate how students spend 
their time and what their involvements are. The Competency scales 
require students to evaluate their competencies in a series of activities. On 
the Occupations scales students indicate preferences toward various 
occupations. The Selfi Estimate scales are self ratings of students’ talents 
and traits. All scales included in the SDII correspond to the six categories 
in Holland’s career development theory. The instrument is presented in an 


,eight-page (8% x 5%) booklet. An occupational classification booklet, The 


Occupations Finder, arranges 465 occupational titles according to the six 


personality types. Each occupational subclass is arranged according to the: 


educational level required by the occupation. Most occupations are 
cross-referenced to the Dicsionary of Occupational Titles. 


*Copyright 1974, by Consulting Psychologists Press, Inc. Adapted by special permission. 
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3. Administration Procedures: The SDiI is amenable to individual or group. 

. administration, Students read the instructions listed on the inside cover of 

the booklet. If administered in a group setting, the test proctor can read 

the instructions aloud. Administration time is approximately 30 minutes. 
Students may omit the occupation code ID on the answer sheet. 


4. Scoring Procedures: Processing is available from RBS, Scores are generated 
by summing preferences within each subscale and ranking the highest 
three scores. Thus, a three letter code which corresponds to the six 
occupational categories is derived for each section. A summary code is 


obtained as a weighted combination of all sales, 
a : 


Bo iScaeing Interpretation and Use: The lideronatatie of the SDII is srnilar 
to the interpretation of other interest inventories. Summary codes 
indicate interests or preferences in relation to Holland’s six categories. 
Results can be used on an individual or group basis. Individual profiles 
may be used in conjunction with The Occupations Finder for counseling .. 

12 purposes so that students can see which types of occupations match t 
interests and competencies. The SDII also can be used to direct students 
toward career exploration sequences which are meaningful to them. In 
addition, the SDII can define career maturity according to the congruency 
between students’ summary codes and the occupations they are consider- 
ing. In the context of group guidance, results can be used by constructing 
exercises that utilize the SDII as a starting point. For example, the SDII 
can be used to explore.the relationships between occupations, activities, 
competencies, self-estimates and occupational clusters, Results also have 
implications for program planning, for instance, revising career explora- 
tion activities on the basis of SDII data. Further information on 
interpretation can be found in the Counselor’s Guide and Professional 
Manual of the Self-Directed Search A pabliahed Ps Consulting Psychologists 

Press, * * 


6. Psychometric Qualities: The validity and reliability of the SDII are — 
inferred from studies available on the SDS. Content. validity has been 
demonstrated by consistency with well-established vocational concepts 
and theories. Studiés have indicated low to moderate predictive validity 
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Sy . 
for variables such as non-academic achievement, vocational aspirations and 
self-ratings. Concurrent validity is‘ demonstrated by correlations with 
instruments such as the Kuder Preference Inventory, the Strong Voca- 
tional Interest Blank, the Bennet Mechanical Comprehension Test and the 
Minnesota Paper Form: Board, The scales have a moderate degree of 
internal consistency with Kuder-Richardson coefficients ranging from .53 
to .88. Summary codes have a high degree of test-retest reliability (.81 to 
.92), although individual scales have somewhat lower coefficients. 


Availability: The SDII, optical-scanning response sheets and processing 
services are available from RBS. The Occupations Finder and technical 
manuafs are available from Consulting Psychologists Press. 
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E. ‘ASSESSMENT OF STUDENT ATTITUDES. TOWARD LEARNING © 
ENVIRONMENTS (ASA) 


Rationale/Objectives of Instrument: This instrument was designed to 
measure secondary school students’ attitudes toward learning environ- 
ments, both traditional and non-traditional. 


' 


Content and Organization: The 26-itém instrument is presented on a 
one-sided optical scaning form. It includes four subscales and an overall 
‘score: Attitude Toward Education in General, Attitude Toward School 
Curriculum, Attitude Toward School Resources, Attitude Toward School 
Counseling and Overall Attitude Toward Learning Environments. Each 
item utilizes a five-point Likert scale with “Strongly Disagree” and 
“Strongly Agree” as poles. 


Administration Procedures: This instrument can be administered in an 
individual or group setting. Students should receive instructions for 
completing optical scanning answer sheets and complete all appropriate 
identification information. The test-administrator should then read the 
instructions printed on the instrument to the students and answer any 
questions that might arise. Administration time is approximately 15 
minutes. 


Scoring Procedures: Processing is available from RBS. Item responses are 
arranged by correct polarity weighted by appropriate scale value and 
summed for each subtest and for the total instrument. 


Scoring Interpretation and Use:- Scores are most easily interpreted as item 
means for each of the subtests and for the total test. Item means close to 
five indicate a very strong positive attitude, whereas item means close to. 
one indicate~a very strong negative attitude. Answer sheets should be 
scanned to determine the extent to which students choose response 
category ‘3’ (‘not sure’’). If there is a high incidence of such responses, 
item means will tend to regress toward the midpoint. Subtest scores can 
be used to describe general satisfaction of program students with various 
elements, while comparisons with control group students ‘can indicate 
areas of program effectiveness. 


19, 
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- Survey published ore RBS. 


x Psychometric Qualities: The ASA has demonstrated high reliability, The ; 


split-half coefficients for subscales range from .56 to .74 with an overall 
coefficient of .80. Item to subscale correlation coefficients range from .46 

to .71 with an average correlation of .58, As an estimate of validity, -. 
subscale scores were compared with staff ratings and percent of agreement 
was calculated. Although interrater reliability was low, the extent of 
agreement ranged from 47 to 72 percent with an average of 56 asi 
thus lending support to validity claims. 


Availability: The ASA and processing services may be obtained from RBS. ' 
The instrument also is included as a subscale within the Student Attitude 
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F. STUDENT ATTITUDE SURVEY (SAS) 


Rationale/Objectives of the Instrument: This instrument is designed to . 


measure secondary school students’ attitudes toward school, work, self 
and others. : 


Content and Organization: The 80-item instrument is presented on a 
two-sided optical scanning form. It contains four separate sections 
including the various subscales described below. 
@ Assessment of Student Attitudes Toward Learning Environments 

— education in general 

— school counseling 

— school curriculum 

_— school resources 

— overall learning environments 
@ Career Attitude Survey 

— career knowledge 

— career planning 
@ Acceptance of Self Scale 
@ Acceptance of Others Scale 


All ftems are presented in.a five-point Likert format’ with “Strongly 


Disagree” and “Strongly Agree’’ as poles. 


Administration Procedures: The instrument can be adrhinistered in an, 


individual or a group setting. Students should receive instructions for 
completing optical scanning answer sheets and complete all appropriate 
identification information. The test administrator should then read the 
instructions printed on the instrument to the students and answer any 
questions that might arise. Administration time is approximately 30 
minutes. 


21 


* 


13 


Scoring Procedures: Preiiessiig’t is ‘evallable fro RBS. Item responses are" 
arranged by correct polarity, weighted: by ea aa scale value and 
summed for each subscale. 


“ye 

Scoring Interpretation and tlie: Scores are most easily interpreted as item 
means for each of the subscales and overall scales. Item means close to five _ 
indicate a very strong positive attitude; whereas item teans close to one 
-indicate a very strong negative attitude. Answer sheets should be scanned 
to determine the extent to which students choose response category sad 
(“not sure”). If there is a high incidence of such responses, means will. 
tend to regress toward the midpoint. Group subtest means can be used to 
describe general attitudes of program students, while comparisons with 
control group students can indicate areas of program effectiveness. 


Peychmetrie Qualities: The SAS has demonstrated a high. degree a 
internal consistency. The average item to total score correlations for the 
major scales were .48, .53, .57. and .50, The split-half reliability’ - 
coefficients were .80, .90, .88 and .70. Two measures of validity were 
employed: agreement of staff and student ratings and scale sensitivity to 
experimental treatment effects. In the first case, despite low interrater — 
reliability, the extent of agreement ranged from 47 to 72 percent with an 
average of 56 percent. The second index of validity indicated the 
instrument’s ability to identify experimental students who had been 
exposed to a program designed to affect the variables which the 
instrument measures in six of seven cases. 


Availability: The Student Attitude Survey and processing services may be 
obtained from RBS. 


G. COMPREHENSIVE TESTS OF BASIC SKILLS (CTBS) 


1. Rationale/Objectives of ie Instrument: The Comprehensive Tests of 
Basic Skills are a series OWtest batteries produced in four overlapping levels — 
with alternate forms (Q and R, S and T). The batteries at each level 
contain tests in four basic skills content areas: Reading, Language, : 
Arithmetic and. Study Skills. Science and Social Studies tests have been 
added to Forms S and T. The CTBS measures student development in four ; : 
process afeas (following Bloom’s classification schema): recognition 
and/or classification, translation, interpretation and analysis. Three 
subtests have been chosen by the RBS evaluation staff for inclusion in the 
instrument ‘package: Reading Comprehension, Arithmetic Concepts and 
Arithmetic Applications, Only Forms Q and R are discussed in this section 
since these have been used exclusively to date. Level: 4 is usually 
appropriate for secondary-age students. Information on.the other subtests iat 
can be found-in the CTBS Test Coardinators’ Handbook. - 19 . 


2. Content and Organization: Alf subtests are included in a booklet format. 
Separate’ answer sheets are available for the Reading and Arithmetic 
subtests. The tests are in multiple-chdice form with four response options. 


The Reading Comprehension test contains 45 items which refer to reading 
passages presented in the form of articles, stories, lettersor poems. Items 
can be classified into the following process categories: paraphrasing ideas; 
identifying the main idea; perceiving relationships; drawing conclusions; 
making inferences; extending interpretation beyond stated information; 
and recognizing the author’s intention. 


The Arithmetic Concepts test contains 30 items based on the number 
system, measurement, algebra, geometry, statistics and logic. The items 
can be classified into the following: process categories: recognition and/or 
application of concepts and techniques; conversion of concepts into other 
forms; comprehension of the interrelationships of numerical concepts; and 
the organization of facts in complex problems. 


The Arithmetic Applications test contains 20 items with emphasis upon 
problem solving. The items address the’ following process categories: 


s 23 


. minth and tenth grade students. Standard scores 
" analyzing student growth and comparing groups of students. Tables to 


| comprehension of probletie: selection of appropriate methods of solution; a ; 


organization of facts in complex problems; and solution. of problems. uy 


. . Administration Procedures: The test shouldbe adsiniidersd” in a ‘ebony’ 


setting. The CTBS Examiner’s. Manual gives complete: instructions for 
administering all tests in a standardized format. Time limits are as follows: 


_ Reading Comprehension test, 30 minutes; Arithmetic Concepts test, 15 


minutes; Arithmetic Applications test, 14 ‘minutes. Approximately 10 
minutes will ‘be required for giving instructions. The test administrator 
should make sure that students answer questions on the correct answer | 
sheet for the appropriate test. 


‘Scoring Procedures: The raw score for each test is the number of items 


answered correctly. Percentile scores, stanines, grade equivalents, or scale 
scores can be derived from conversion tables provided by CTB/McGraw- 


Hill. The CTBS can be scored either by hand or machine. Keys and scoring 


stencils for hand scoring are available from CTB/McGraw-Hill. Appropri- — 
ate answer sheets can be machine scored either by CTB/McGraw-Hill or 
RBS. 


Scoring Interpretation and Use: The higher the raw score on the CTBS, ° 
the greater a student’s knowledge in the area. Grade equivalent scores are 
derived from. the average raw score in a representative sample of all pupils 
in that grade level in American schools. Grade equivalents are easily 
intérpretable in a descriptive , sense but are not recommended for 
comparative analyses. Percentile scores indicate the percentage of students 
in a group who are below the given score. The reference group may be the. 
nationwide sample reported in the CTBS manual or norms constructed on 
the local level. A single scale of standard scores for use with all levels and 


~ any form of the CTBS has been developed and reported by CTB/McGraw- 
. Hill. This Expanded Standard Scale Score ran: 


rom approximately 100 - 
to 900 and is based on scores from the national staXdardization sample of _ 
é récommended for 


equate scores of all forms of the CTBS are available from CTB/McGraw- 


Hill. 2 4 


Results of the CTBS may be used to indicate student peowiht in basic skills - 
or to compare performances between program students and students in” 
local control groups or in the national standardization sample. Scores can 
be analyzed for individuals or for groups of students. _ 


Psychometric Qualities: A comprehensive description of the development 

_of ‘the CTBS is given in the CTBS Technical Report. Norms were 
established through tryouts with a large sample of students representative 
of the population of American secondary school students. ‘Items were 
selected on the basis of difficulty, discrimination and reliability. The 
difficulty indices for items within each test ranged from .25.to .90 with a 
mean difficulty of approximately .60. All items demonstrated biserial 
correlation coefficients greater than .30, Reliability coefficients (KR-20) 
for the Level 4 Reading Comprehension test, Arithmetic Concepts test .. 
and the Arithmetic Applications test are .89, .86, and .84, respectively. 
Correlations between Forms Q and R (testing interval of six weeks) for 
the three subtests range from .76 to .84. Evidence ‘of content and 
congruent validity is noted in the CTBS Technical Report. 


Availability: CTBS test booklets, answer sheets and accessories can be 
obtained from CTB/McGraw-Hill, Processing for Digitek answer sheets is _ 
available from RBS. 
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H. STUDENT OPINION SURVEY (SOS) 


Rationale/Objectives of the Instrument: This instrument is designed to 
measure student .opinions concerning career education programs similar to 


- RBS Career Education, 


Content and Organization: The instrument contains 20 items incorpérat- 
ing a five-point interval scale and includes three: open-ended items: The 
items assess attitudes toward various program elements, opinions on 
program benefits and comparisons of the program relative to standard 


curricular offerings. The instrument is presented in. an eight-page 
(8% x 5%) booklet. 


Administration Procedures: The instrument can be administered to groups 
or to individuals. If confidentiality of responses is a concern, students may 
omit identification information on the first. page. Administration time is 


‘approximately 15 minutes. 


Scoring Procedures: The Student Opinion Survey must be scored by hand. 
Response category frequencies and percentages for individual items are 
most appropriate. Item means also may be helpful. An overall instrument 
score is not appropriate. Open-ended responses can be listed verbatim 
(eliminating personal references), and a categorization for summarizing 
and reporting responses can be devised. 


Scoring Interpretation and Use: Item means close to five indicate a strong 
positive rating; a mean close to one indicates a strong negative rating. 
Frequency distributions of response categories as well as item means 
should be examined for further description of general student opinion. 
Rate of response should be considered to determine the generalizability of 
results. Feedback from the SOS can assist in future program planning 
efforts. 


Psychometric Qualities: Psychometric analyses of previous versions of the 
SOS indicate a high degree of internal consistency (KR-20 of .99) and 
homogeneity of factors. Based on these analyses several items have been 
deleted and further psychometric studies are being conducted on the 
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- revised instrument. Content. validity and usefulness ofthe instrument were. oe 
established by a group of career education program evaluators. The 
instrument has been revised over a four-year period, 


- 


Ta Availability: The, Student Opinion Survey ‘and processing services can abe ae. 
obtained from RBS. ; 
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“IL PARENT OPINION SURVEY (POS) = 


‘ 
1. 


Rationale/Objectives.- of Instrument: This. instrument is designed . to 
measure opinions and attitudes of the parents: ‘of students enrolled in 
career education progtams similar to RBS Career Education, 


Content and Organization: The instrument contains 13 items incorporat-_ 
ing a five-point interval scale and includes two open-ended items. The’ 
items assess attitudes toward various program’ elements, opinions on 


program benefits and comparisons of the program relative to standard 
curricular offerings. The instrument is presented in a four? ive x 544) 
booklet. 


‘ 


Administration Procedures: The method of survey administration will: . 


depend upon the local context. This, in turn, will influence, to a large 


extent, the rate of response. In most cases, the surveys have been mailed . . 


to parents with a pre-paid return envelope. A cover letter indicating the. 
purpose of the survey and the return deadline and emphasizing the 
importance of responses should-be included in such a mailing. It may be 


desirable to follow up unreturned surveys by telephone. Alternatives to 
mailing include administering the instrument at a parent-teacher meeting | 


or sending the surveys home with students. All methods have advantages 


and: disadvantages; factors such as*cost, required effort and rate of ° 


response should be weighed. Administration time is approximately 10 
minutes, 


Scoring Procedures: The Parent Opinion Survey must be scored by hand. 
Response category frequencies and percentages for individual items are 
most appropriate. Item means also may be helpful. An overall instrument 
score is not’ appropriate. Open-ended responses can be listed verbatim 
(eliminating personal references), and categorization schema for summa- 
rizing and reporting responses can be devised. 


Scoring Interpretation and Use: Item means close to five indicate a strong 
positive rating; a mean close to one indicates a strong negative rating. 


».Frequency---distributions--of--response~categories--as--well--as-item~means 


should be examined to obtain further descriptions of general parent 
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‘ available from RBS. 


‘opinion. Rate ‘of response. death be peinained: to date tee. 
a generalizability of results. A special consideration in reviewing POS results. 
__ is that parents may not feel that they have. endugh information about the 


career education program to. rate its quality and/or effects. This view may _ 
be.reflected in a low rate of ‘response or a high number of responses in. 
gategory “3”, Such a finding would indicate the need for..a parent’ 


orientation program and a system of information. dissemination. Other... 


feedback from the POS can assist in future program planning efforts. 


Psychometric Qualities: Psychometric analyses of previous versions ofthe 


‘ POS indicate“a high degree of internal consistency (KR-20 of .93) and : 


homogeneity of factors. Based on these analyses, several items have been 


deleted and further psychometric studies are being conducted on: the . 


revised instrament. ‘High degrees of reliability ; and construct validity are. 
expected, -as: was the case with previous versions of the POS. Content 


validity and usefilness of the instrument were established. by a group. “a 


career education prograth evaluators. The idbtrument has been revised over 
a four-year period. aut 


4 


Availability: The Pabent, Opinion Survey and fpccsny services are - 


1, ‘ 


2. 


‘COMMUNITY PARTICIPANT OPINION SURVEY (cros} 


’ 


Rationale /Objectives of the Instrument: This js instrument is designed to. 


measure community participant opinions ° ‘concerning educational pro- 


grams similar to RBS Career Education. . 


Content and Organization: The instrument contains 13 items incorporat- 
.ing a five-point interval scale and includes two open-ended items. The 


items assess attitudes towards various program elements, opinions on. 


program benefits and comparisons of the program relative to standard 
curricular offerings. The instrument is ‘presented i ina sak (8% x 5%) 
booklet. 


Administration. Procedures: Administration has been most successful 


when the project field representative delivers the survey to &mmunity 


site respondents and collects it upon its completion. This approach assures - 


a high response rate. Another method has been to mail surveys to 
community participants and include a pre-paid return envelope. In this 
case a’follow-up by telephone often is necessary. Administration time is 
approximately 10 minutes. : 


Scoring Procedures: The Community Participant Opinion Survey must be 
scored by hand. Response category frequencies and percentages for 
individual items are most appropriate. Item means also may be helpful. 
An overall instrument score is not appropriate. Open-ended responses 
can be listed verbatim (eliminating personal references), and categoriza- 
tion schema for summarizing and reporting responses can be devised. 

. ‘ f 
Scoring Interpretation and Use: Item means close to five indicate a strong 
positive rating; a mean close to one indicates a strong negative rating. 


Frequency distributions of response categories as well as item means - 


should ‘be examined for further description of general community 
participant opinion. Negative results can alert program staff to possible 


resource site dissatisfaction, while positive results can ae very helpful i in 
"gaining support of other community agencies, : 
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Prvehinelete Qiilties: No formal study of psychometric qualities Wes ey 
been conducted, Content. validity has been established through : reviews MY 
career education program evaluators and staff members, ‘J 


Availability: The Community Participant Opinion Survey and processing 
services can be obtained from RBS. ; , es 
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_INTERRELATIONSHIPS AMONG INSTRUMENTS > * 


The figures presented below represent ‘the ‘ntercorrelations among the 
instruments discussed in this guide.* These statistics were calculated in order to 
provide further information which may ‘be helpful in the instrument selection’ 
process. The figures show that the instruments measure relatively independent ° 
phenomena. The correlations among tests are fairly low; the relationships Among . 
subtests within instruments are generally higher. 


Instruments 1 2 3 4 5 6 7 8 9 
~ 4, SAS — Learning Environments. 1.00 . , ’ 
2. SAS — Careers nf 9 oe | rh 4 ; 
3, SAS— Self-Concept at as | Bs phe | 
4. SAS—Others-Concept, 2132S | _ 29 - 
5. CTBS — Reading Comprehension = -.11 18 +20 -.29 : : haa = 
6. CTBS— Arithmetic Concepts 06.08 -.15 -.20 .35 ; 
7. CTBS— Arithmetic Applications -.0B -.01 -.06 -19 .28 .70 
8. CMI — Career Knowledge 04 -24°+.23 +34 49 44 41 :. 
9. CMI — Career Planning - i‘ 06-17. 23 +29 41 49 42. «611.00 
_ N=201 . | 
; 4 


"The ACD has not been included in-these Yelational analyses-to-date.-s 


e 


SUMMARY OF INSTRUMENTS AND SERVICES AVAILABILITY: . 


CTB — 


Research for Better Schools, Inc. 


Instruments 

A> SDQ RBS . 
B. CMI: : CTB 
C. SDII RBS 
D. ACD HM 
E. ASA RBS 
F. SAS RBS: 
G. CTBS CTB 
H, ee RBS 
I; POS RBS 
J.) CPOS RBS 

RBS — __ Evaluation Services ° 


4 


1700 Market Street, Suite 1700 
, Philadelphia, Pennsylvania 19103 


CTB/McGraw-Hill 


Order Service Center 


Del Monte Research Park 
Monterey, California 93940 


RBS RBS 
CTB or RBS CTB / 
RBS CPP 
HM HM 
RBS RBS 
RBS RBS 3 l 7 
CTB or RBS CTB . 
RBS . RBS : 
RBS RBS 
CRs RBS 
CPP — Consulting Rsychologists Press 
577 College Avenue 
Palo Alto, California 94306 
HM— Houghton Mifflin 
Regional Sales Office 
Pennington — Hopewell Rd. 
Hopewell, New Jersgy 08525 
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Scoring Manuals, Etc. 


»> 
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| CONCLUDING NOTES 


The selection of appropriate instruments to match: program objectives is a - 
. crucial step in the development of an evaluation plan, Factors such as instrument 
content, administration time and procedures, form and utility of scores and 
psychometric qualities should be considered carefully. This guide has discussed these 
factors specifically in relation to instruments which are recommended for 
consideration in the evaluation of career education programs, p 
A computer-linked scoring service is available through RBS Evaluation Services 
for. each instrument described in the guide.’ This service. produces standard 
80-column keypunched computer cards for users interested in performing their own 
data analyses, Printed: profiles can also. be prepared. Individual profiles can be 
generated for each student, listing student. scores for every itistrument used and the. 
_ position of the student relative to local or national norms. Group profiles summarize 
results across students and provide average scores or “rankings and standard 
deviations for each instrument. Additional services can be tailored to the needs of 
users; technical assistance in all aspects of the evaluation of career education. 
programs is available. Research for Better Schools, Inc. can provide further 
information on the availability of instruments and services discussed in this guide. 
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The publication of RBS CAREER EDUCATION: INSTRUMENT SERVICE GUIDE 
has been funded in part by the Vocational Education Act (P.L. 8910, Title IV)‘ as 
amended in 1968, in cooperation with the National Institute of Education, undet | 
research contract #NE-C-004-0011. The. opinions expressed in this publication do 
not necessarily reflect the position or policy of the National Institute of Education, 
and no official endorsement by the National Institute of Education ‘should be 
inferred. 


Research for Better Schools, Inc. 
1700 Market Street, Suite 1700 © ' 


Philadelphia, Pennsylvania 19103 », 
Robert G. Scanlon, Executive Director 


